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Abstract 


Purpose:  Explore  effects  of  implementing  a  professional  practice  model  (PPM)  on  optimum 
care  indices  in  two  military  ambulatory  clinics. 

Design;  This  2x3  fixed  factorial  design  examined  32  access,  continuity,  staff  and  patient 
satisfaction,  and  quality  care  indices  in  Family  (FM)  and  Internal  Medicine  (IM)  clinics  across 
three  time  periods. 

Sample/Methods;  Nursing  staff  (n=42)  and  patients  (n=1220)  were  recruited  using  non- 
purposive  sampling  for  the  satisfaction  questionnaires.  Aggregated  quality  care,  access,  and 
continuity  metrics  from  each  clinic  and  selected  questions  from  the  Army  Provider  Level 
Satisfaction  Survey  (n=2834)  and  the  Interactive  Customer  Evaluation  (n=4275)  were  analyzed 
using  descriptive  and  inferential  statistics.  Qualitative  data  from  questionnaires  were  analyzed 
using  content  analysis,  identifying  points  of  convergence,  divergence,  and  complementarity  with 
quantitative  data. 

Analysis/Findings;  Communication  (p=.004),  access/convenience  (p=.001),  see  provider  when 
needed  (p=.039)  and  overall  satisfaction  (p=.015)  improved  over  time.  Patient  satisfaction  was 
more  favorable  in  FM  than  IM  (Wilk's  A  =  .982,  p=  .001).  Staff  satisfaction  improved  in  FM 
(p<.05)  only;  RN/MD  relationships  improved  {F  (2,  85)  =  19.2,  p  <  .05)  in  both  clinics.  Few 
quality  care  metrics  improved  in  either  FM  (n=2)  or  IM  (n=3).  Qualitatively,  staff  identified 
significant  work  turbulence:  frequent  changes,  lack  of  resources,  ineffective  leadership 
communication,  management  style,  staffing,  and  practice  constraints  as  issues.  Furthermore, 
Patient  Centered  Medical  Home  (PCMH)  implementation  eclipsed  PPMs,  as  nursing  staff  felt 
excluded  from  decision-making.  Staff  dissatisfaction  was  mirrored  in  the  patient  comments,  as 
patients  felt  rushed,  commenting  that  staff  needed  more  help. 

Military  Nursing  implications;  Conduct  research  and  EBP  projects  to  refine  nurse-sensitive, 
outpatient  outcome  measures,  develop  a  staff  satisfaction  instrument  for  use  with  multilevel 
nursing  staff  and  assess  effects  of  PCMH  teams  on  nurse-sensitive  metrics.  Educate  leadership 
and  staff  regarding  change  management,  effective  communication,  nurses  practicing  at  the  top 
of  their  education  and  training,  and  interdisciplinary  collaboration  in  the  spirit  of  the  IQM  report. 


Word  count  without  headings:  296 
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TSNRP  Research  Priorities  that  Study  or  Project  Addresses 


Primary  Priority 

Force  Health  Protection: 

□  Fit  and  ready  force 

□  Deploy  with  and  care  for  the  warrior 

□  Care  for  all  entrusted  to  our  care 

Nursing  Competencies  and 
Practice: 

□  Patient  outcomes 

Kl  Quality  and  safety 

^  Translate  research  into  practice/evidence-based 
practice 

IKI  Clinical  excellence 

□  Knowledge  management 

□  Education  and  training 

Leadership,  Ethics,  and 
Mentoring: 

□  Health  policy 

□  Recruitment  and  retention 

□  Preparing  tomorrow’s  leaders 

□  Care  of  the  caregiver 

Other: 

□ 

Secondary  Priority 

Force  Health  Protection: 

□  Fit  and  ready  force 

□  Deploy  with  and  care  for  the  warrior 

□  Care  for  all  entrusted  to  our  care 

Nursing  Competencies  and 
Practice: 

^  Patient  outcomes 

O  Quality  and  safety 

□  Translate  research  into  practice/evidence-based 
practice 

□  Clinical  excellence 

□  Knowledge  management 

□  Education  and  training 

Leadership,  Ethics,  and 
Mentoring: 

□  Health  policy 

□  Recruitment  and  retention 

□  Preparing  tomorrow’s  leaders 

□  Care  of  the  caregiver 

Other: 

□ 
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Progress  Towards  Achievement  of  Specific  Aims  of  the  Study  or  Project 

The  purpose  of  this  evidence-based  program  was  to  explore  effects  of  implementing  an 
evidence-based  professional  practice  model  (PPM)  on  nurse  satisfaction  and  nurse-sensitive 
indicators  of  quality  care  in  a  military  ambulatory  setting.  Professional  practice  models  provide 
the  scaffolding  to  support  healthy  work  cultures  that  facilitate  the  design  and  delivery  of  optimal 
care  and  patient  outcomes  (American  Association  of  Colleges  of  Nursing  [AACN],  2002;  Hoffart 
&  Woods,  1996;  Girard,  Linton,  &  Besner,  2005).  The  Army  Nurse  Corps  (ANC)  embedded  a 
PPM  in  its  philosophy  of  care,  the  Patient  CaringTouch  System  (PCTS)  (Horoho,  2011). 
Essential  elements  of  the  PCTS  PPM  identified  in  the  literature  and  by  subject  matter  experts 
included:  professional  nursing  values  and  ethics,  interdisciplinary  collaboration,  differentiated 
practice,  shared  governance,  and  employee  recognition  (Table  1).  Family  (FM)  and  Internal 
Medicine  (IM)  outpatient  clinics  implemented  the  PPM  guided  by  the  Iowa  Model  of  Evidence- 
Based  Practice  over  a  three-year  period  2009-2012.  During  implementation,  outpatient  clinics 
Army-wide  transitioned  to  patient  centered  medical  homes  (PCMH).  The  PCMH  are  provider- 
focused  while  the  PPM  is  nursing  centric.  Specific  aims  of  this  project  were: 

1 .  Evaluate  levels  of  nursing  satisfaction  (Essentials  of  Magnetism-ll  [EOM-II®])  and  patient 
satisfaction  (Patient  Satisfaction  18-item  Questionnaire  [PSQ-18],  Army  Provider  Level 
Satisfaction  Survey  [APLSS],  and  the  Interactive  Customer  Evaluation  [ICE])  in  ambulatory 
care  clinics  at  Tripler  Army  Medical  Center  (TAMC)  following  implementation  of  a  PPM. 

2.  Evaluate  the  effects  of  implementation  of  a  professional  practice  model  on  baseline 
measures  of  patient  satisfaction,  nurse-sensitive  indicators  of  quality  care  as  measured  by 
selected  National  Committee  for  Quality  Assurance  health  effectiveness  and  data 
information  set  (HEDIS),  access  to  care  APLSS  measures  (Time  between  scheduling  and 
visit,  wait  time  in  clinic,  and  see  provider  when  needed),  and  continuity  (percent  of  time  seen 
by  Primary  Care  Manager  [PCM]  by  name  and  by  team)  in  ambulatory  care  clinics. 

3.  Evaluate  the  relationship  between  staff  satisfaction  (EQM  II®)  and  patient  satisfaction  (PSQ- 
18,  APLSS,  &  ICE)  and  quality  of  care  (HEDIS  &  PCM  by  name/team)  outcomes  following 
the  implementation  of  the  PPM. 

Staff  Satisfaction 

The  1984  Nursing  Work  Index  questionnaire  (NWI)  (Schmalenberg  &  Kramer,  2008) 
measures  work  factors  which  influence  nurse  satisfaction  and  are  linked  to  patient  outcomes 
(Aiken  et  al.,  2008).  Revisions  to  the  NWI  resulted  in  the  EQM-II®.  The  EQM  II®  has  eight 
subscales:  Clinical  autonomy,  working  with  other  nurses  who  are  clinically  competent, 
collegial/collaborative  RN/MD  relationships,  perceived  support  for  education,  control  of  nursing 
practice,  perceived  nurse  manager  support,  perceived  adequacy  of  staffing,  and  patient- 
centered  culture/values.  The  total  EQM  II®  score  (sum  of  weighted  subscales)  is  termed 
Professional  Job  Satisfaction  and  indicates  the  extent  to  which  staff  nurses  report/confirm  a 
healthy  work  environment  (appendix  A).  The  total  weighted  EQM  II®  score,  professional  job 
satisfaction,  will  be  the  primary  focus  of  the  analysis  as  recommended  by  the  authors  and 
reported  in  the  literature  (Kramer  and  Schmalenberg,  personal  communication  November  2013; 
Newhouse,  Morlock,  Pronovost,  Colantuoni,  &  Johantgen,  2009).  Scores  less  than  290  indicate 
a  work  environment  that  needs  improvement  (WENI),  290-31 1  indicates  a  healthy  work 
environment  (HWE),  and  over  31 1  indicates  a  very  healthy  work  environment  (VHWE)  (Kramer 
and  Schmalenberg,  personal  communication  November  2013). 
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Table  1 

Army  Nurse  Corps  Professional  Practice  Model  (PCTS)  Tenets  Supported  by  the  Professional 
Practice  Model  (PPM) 


PCTS  Tenets 


Nursing  Ethos 
Accountability 


Patient  &  Family  Centered 
Communication 


Evidence  Based  Practice 
Healthy  work  environment; 
Skill  building;  Staff 
development 

Leadership 


Support 

Healthy  work  environment 


Professional  Practice 
Elements  in  the  literature 
Professional  nursing 
philosophies  and  values 


Interdisciplinary 

collaboration 


Differentiated  nursing 
practice/care  delivery 
system 


Management  decision 
making 

Recognition  of  nurses’ 
contributions 


Professional  Practice  Model  Tenets 


Theory  of  Caring,  accountability,  and 
professional  ethics;  critical  thinking, 
empowerment,  shared  vision  with 
organizational  mission,  values,  and 
vision,  autonomy  over  nursing  practice 
Interdisciplinary  collaborative 
relationships,  nurse  accountable  for 
plan  of  care  in  partnership  with  patients, 
families,  and  other  members  of  the 
healthcare  team;  patient  centered 
teamwork  through  communication. 

Using  evidence-based  approach  to 
improve  standards  of  care;  professional 
responsibilities  commensurate  with 
licensure  and  scope  of  practice  and 
skills 

Shared  governance,  empowerment  of 
UPCs,  coordination  with  Results 
Council 

Supports  public  and  individual 
recognition  for  progress  towards  and 
achieving  goals  Supports  professional 
development 


For  the  EOM  II®  scales  the  highest  intercorrelations  were  found  for:  (1 )  Clinically  competent 
peers  and  values:  r=  .75;  (2)  Supportive  nurse  manager  and  values:  r=  .75;  (3)  Clinically 
competent  peers  and  supportive  nurse  manager:  r  =  .67  and  (4)  Support  for  education  and 
clinical  autonomy:  r  =  .66.  Demographics  for  the  EOM  II®  revealed  that  94.9%  worked  8  to1 0 
hour  day  shifts  whereas  5.1%  worked  8  to  10  hour  in  evening  shifts.  For  highest  level  of 
education,  42.9%  responded  'other' ,  followed  by  22.6%  with  Baccalaureate  (BSN).  For  years  of 
experience  the  mean  was  8.3  years  (SD  =  6.77,  Median  =  6). 

Patient  Satisfaction  Questionnaires 

The  PSQ-18  for  ambulatory  care  has  well  established  validity  and  reliability  with  acceptable 
internal  consistency  for  all  subscales  (Marshall  &  Hays,  1994).  The  seven  subscales  measured 
general  satisfaction  (items  3  and  7);  technical  quality  (items  2,4,6,  and  14);  interpersonal 
manner  (items  1 0  and  1 1 );  communication  (items  1  and  1 3);  financial  aspects  (items  5  and  7); 
time  spent  with  provider  (items  12  and  15);  accessibility  and  convenience  (items  8,  9,  16,  and 
18).  Higher  scores  (range  1  -  5)  reflect  satisfaction  with  medical  care.  Deleted  items  on  the 
PSQ-18  were  not  applicable  to  a  military  setting.  For  example,  question  seven  “I  have  to  pay 
more  for  my  medical  care  than  I  can  afford”  was  removed  as  there  are  no  out-of-pocket 
expenses  for  beneficiaries  seen  at  a  military  treatment  facility.  Questions  mentioning  “doctor” 
were  modified  to  “provider”  as  the  provider  level  was  not  a  variable  in  this  study.  Qne  open- 
ended  item  “Please  add  any  comments  you  feel  will  help  us  improve  our  service  to  you”,  one 
demographic  item,  and  three  yes/no  items  were  added  to  the  survey  (appendix  A). 
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For  the  PSQ-18,  52.3%  (n  =  638)  were  from  IM  and  47.7%  (n  =  582)  from  FM.  Regarding 
time,  30.6%  (n  =  373)  were  observed  at  time  one,  33.2%  (n  =  405)  at  time  two,  and  36.2%  (n  = 
442)  were  observed  at  time  three.  For  the  total  sample,  43.7%  were  seen  1-2  times  in  the  clinic 
the  last  6  months,  69.9%  knew  what  healthcare  team  they  are  assigned  to,  89%  knew  their 
primary  care  providers  name,  and  26.3%  knew  their  primary  nurses  name. 

The  highest  coefficient  alpha  for  the  PSQ-18  subscales  was  obtained  for  Technical  Quality 
(a  =  .709)  followed  by  Accessibility  and  Convenience  (a  =  .689).  The  rest  were  <  .623  which 
warrants  some  concern  about  the  psychometric  properties  of  these  subscales  in  this  sample 
population. 

Army  Provider  Level  Satisfaction  Survey  (APLSS). 

This  24-item  survey  is  managed  by  the  Qffice  of  the  Surgeon  General  and  sent  to  randomly 
selected  patients  within  48  hours  of  their  outpatient  visit  throughout  the  Army  Medical 
Department  (AMEDD).  Qnly  those  questions  addressing  satisfaction  with  staff,  access,  and 
continuity  (n=8)  at  TAMC  were  included  in  this  analysis.  For  example  item  1 1  “The  amount  of 
time  from  when  you  made  the  appointment  until  you  actually  saw  the  healthcare  provider”  was 
used  to  evaluate  access.  The  5-item  verbal  response  scale  ranges  from  strongly  disagree  to 
strongly  agree.  Item  13  “courtesy  and  helpfulness  of  the  staff  during  this  visit”  addresses 
satisfaction.  Response  scale  for  these  items  are  a  5-item  likert  scale  from  poor  to  excellent 
(appendix  A).  Data  from  2,834  APLSS  surveys  were  analyzed:  37.5%  (FM  n=381;  IM  n=682)  at 
time  one,  31  %  (FM  n=309;  IM  n=569)  at  time  two,  and  31 .5%  (FM  n=229;  IM  n=664)  at  time 
three. 

Interactive  Customer  Evaluation  (ICE)  Comment  Card. 

This  Army-wide  16-item  general  customer  satisfaction  survey  has  an  online  or  pen  and 
paper  option.  Questions  are  tailored  to  the  unit  and  results  reported  to  department  leaders.  The 
study  team  had  no  input  with  regards  to  the  distribution  of  this  survey  as  the  focus  was  on 
professional  not  business  practices.  Satisfaction  with  care  is  addressed  with  items  such  as 
“Employee/staff  attitude”  and  “Access  to  medical  care”  with  a  6-item  likert  type  response  scale 
ranging  from  N/A  to  excellent  (appendix  A).  Data  from  4,272  surveys  were  analyzed  by  group 
(FM  n=3595,  84.15%;  IM  n=677,  15.85%)  and  time  (T-1  n=1968,  46.07%;  T-2  n=1209,  28.3%; 
T-3  n=1095,  25.63%).  Survey  response  rate  by  clinic  differed  significantly  (FM  T-1  n=1350, 
37.55%;  T-2  n=1162,  32.32%;  T-3  n=1083,  30.13%;  IM  T-1  n=618,  91.29%;  T-2  n=47,  6.94%; 
T-3  n=12,  1 .77%).  Response  rate  also  varied  by  question.  For  example,  overall  satisfaction  at 
T-1  for  FM  was  n=1255  and  IM  n=284  whereas  at  T-1  for  the  question  regarding  staff  attitude 
n=1350  in  FM  and  n=617  for  IM. 

Quality  and  access  measures. 

Measures  are  reported  in  standard  format  across  all  military  treatment  facilities  and  include 
the  HEDIS  measures,  percent  of  time  seen  by  primary  care  manager  (PCM),  and  percent  of 
time  see  by  PCM  team.  Three  questions  from  the  APLSS  data,  time  between  scheduling  and 
visit,  wait  time  in  clinic,  and  see  provider  when  needed,  comprised  the  access  to  care 
measures.  Between  notification  of  funding  and  the  data  collection  time  periods,  two  access 
measures,  calls  answered  within  90  seconds  and  call  abandonment  rates,  and  one  HEDIS 
measure,  pneumococcal  immunization,  were  no  longer  tracked  and  therefore  not  included  in  the 
study.  Metrics  for  HEDIS  measures  were  evaluated  by  the  number  of  eligible  persons  that 
received  the  treatment.  For  example,  of  those  eligible  for  a  mammogram,  how  many  received  a 
mammogram?  Number  of  eligible  patients  varied  by  metric  as  noted  in  Table  2. 
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Table  2 


Number  of  Eligible  Patients  by  HEDIS  Metric,  Clinic,  and  Time 


Metric 

Famiiy  Medicine 
T-1  T-2 

T-3 

Ciinic 

Totai 

Internai  Medicine 

T-1  T-2  T-3 

Ciinic 

Totai 

Totai 

Mammogram 

955 

929 

797 

2681 

1629 

1638 

1350 

4617 

7298 

Long  term 

asthma 

medication 

86 

90 

79 

255 

195 

61 

90 

346 

601 

Hemogiobin 

ale 

258 

258 

210 

726 

841 

817 

520 

2178 

2904 

A1c<  9 

220 

220 

178 

618 

745 

727 

442 

1914 

2532 

Low-density 

iipoprotein 
ievei  <100 

157 

158 

108 

423 

598 

559 

331 

1488 

1911 

Cervicai 

screening 

2295 

2228 

2083 

6606 

2251 

2160 

2121 

6532 

13138 

Coiorectai 

screening 

912 

941 

698 

2551 

2112 

2184 

1543 

5839 

8390 

Chlamydia 

snrfifininn 

411 

360 

317 

1088 

288 

228 

160 

676 

1764 

Aduit 

pneumococcai 

immunization 

241 

241 

N/A 

482 

1823 

1714 

N/A 

3537 

4019 

Qualitative  data. 

Qualitative  data  was  limited  in  that  respondents  were  not  available  for  face-to-face 
interviews.  Each  instrument  included  an  open-ended  question  asking  respondents  to  provide 
comments  that  would  help  the  clinics  improve  the  care  provided.  Comments  from  the  APLSS 
and  ICE  questionnaires  were  not  available.  Written  comments  from  staff  (EOM  II®)  and  patients 
(PSQ-18)  were  analyzed  using  content  analysis  (Elo  &  Kyngas  2007).  The  unit  of  analysis  was 
individual  sentences,  as  comments  varied  in  length  from  one  sentence  to  a  full  paragraph.  After 
reading  through  all  sentences  several  times,  data  abstraction  proceeded  by  each  team  member 
individually  coding  data  by  group,  clinic,  and  time.  The  PI  compiled  results  using  Atlas  ti  version 
5.2.  The  PI  also  constructed  definitions  for  the  categories  using  literary  data  and  memos  written 
by  team  members  about  various  codes.  For  example,  autonomy  was  defined  as  perceived  self- 
control  over  practice  based  on  terms  used  by  the  authors  of  the  EOM  II®  subscale  (Kramer, 
Schmalenberg,  Maguire,  Brewer,  Burke,  Chielewski  et  al,  2008).  In  a  series  of  meetings  over 
several  months,  team  members  revised  definitions  as  needed  and  condensed  over  162  codes 
into  12  categories  and  two  themes  using  the  definitions  as  a  guide  (appendix  B).  Then  the  PI 
assessed  qualitative  data  for  points  of  convergence,  divergence,  and/or  complementarity  with 
quantitative  data.  Complementarity  refers  to  the  use  of  different  data  sources  to  obtain  alternate 
or  overlapping  views  of  phenomena.  Convergence  refers  to  arriving  at  shared  conclusions  from 
data  collected  by  the  different  methods  such  as  definitions  of  constructs  and  confirmation  or 
strengthening  of  results  thereby  increasing  the  validity  of  outcomes  (Johnson,  Onwuegbuzie,  & 
Turner,  2007;  Tashakkori  &  Teddlie,  2003).  Almost  all  categories  were  bidirectional  in  that 
positive  and  negative  comments  were  applicable  to  each  category.  Staff  and  patient  categories 
included:  access,  communication,  lack  of  resources,  nonclinical  issues,  quality  of  care,  and 
satisfaction  with  care/work  environment.  Staff  specific  categories  were:  autonomy,  change. 
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communication,  leadership,  and  professional  practice.  Continuity  was  specific  to  patients.  The 
overarching  theme  for  staff  was  satisfaction.  For  patients,  the  theme  was  quality  of  care  (table 
3). 

Table  3. 

Staff  and  Patient  Qualitative  Categories  and  Themes 


Staff  Theme:  Satisfaction 

Patient  Theme:  Qualitv  of  Care 

Category  with 
exemplar 

Staff  attitude,  perception,  or 
feeling  towards  work 
environment  or  experience 
(Kalisch,  2010;Peltier, 
2009;Schmaleberg  2008) 

Patient  perception  of 
interpersonal  and  technical 
aspects  of  care  including  safety, 
effectiveness,  efficiency, 
timeliness,  equitable,  patient 
centered  (lOM,  2001) 

Internal 

Family 

Internal  Family 

Medicine 

Medicine 

Medicine  Medicine 

T-1  T-2 

T-3 

T-1  T-2  T-3 

T-1  T-2  T-3  T-1  T-2  T-3 

Access:  Appointments  are 
not  given  adequate  time 
Autonomy:  \Ne  have  a  lot  of 
independence  and 
autonomy  in  our  roles 
Change:  Too  many 
changes  in  past  six  months 
Communication: 

X 

X 

X 

X  X  X  X  X  X 

X 

X 

X 

X 

X 

Communication  about 

X  X 

X 

X  X  X  X  X  X 

rescheduling  is  very  poor 
Continuity:  Not  able  to  see 
same  person  every  time 

X  X  X  X  X  X 

Lack  of  resources:  Please 
enable  our  providers  what 

X  X 

XXX 

X 

they  need 

Leadership:  Whoe ver  is  in 
charge  in  this  clinic  needs 
to  be  employee  of  the 
month 

X  X 

X 

X 

Non-clinical:  Parking! 
Professional  practice: 

X 

X 

X  X  X  X  X  X 

We’ve  become  full  time 

X  X 

X 

X 

telephone  operators 

Quality  of  care:  The  nurse 
taking  my  vitals  was 
courteous  and 

X  X 

X  X  X  X  X  X 

knowledgeable 

Satisfaction  w  care/work: 

Everyone  is  doing  great 

Give  them  a  raise!! 

X  X 

X 

X 

X  X  X  X  X  X 
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Aim  one 

Evaluate  levels  of  nursing  and  patient  satisfaction  in  ambulatory  care  clinics  following 
implementation  of  a  PPM. 

Staff  Satisfaction.  Quantitative  data  (appendix  C)  indicated  that  mean  professional  job 
satisfaction  scores  for  FM  clinic  were  higher  at  T3  and  higher  when  compared  with  IM  clinic. 
Differences  were  not  significant  across  clinics  by  time.  However,  EOM  II®  scores  indicated  that 
FM  clinic  moved  from  a  “work  environment  needing  improvement”  (m=252.89)  to  a  “healthy 
work  environment”  (m=308.9)  at  T-2,  progressing  to  a  “very  healthy  work  environment” 
(m=335.13)  at  T-3.  internal  medicine  clinic  scores  indicated  a  “healthy  work  environment”  at  T-1 
(m=290.81)  and  T-2  (m=297.9),  but  dropped  to  a  “work  environment  needing  improvement” 

(m=  263.79)  by  T-3.  However,  data  must  be  interpreted  with  caution  due  to  low  sample  size. 

Although  staff  satisfaction  should  be  gauged  on  the  composite  professional  job  satisfaction 
score,  summary  for  two-way  ANOVA  (group  by  time)  showed  that  subscales  of  the  EOM 
generally  trended  in  the  same  direction  as  the  composite  scores  as  follows: 

RN/MD  Reiationships.  Significance  was  not  obtained  for  the  two-way  interaction:  F  (2,  85)  = 
2.55,  p  =  .084  (partial  =  .057).  A  significant  main  effect  was  obtained  for  time:  F  (2,  85)  = 

19.2,  p  <  .05  (partial  =  .311,  which  is  a  large  effect  size,  meaning  that  31.1%  of  the  variability 
in  the  outcome  is  attributable  to  differences  between  the  three  time  periods).  Collapsing  across 
groups,  the  12  month  group  had  the  highest  mean  {M  =  48.68,  SD  =  6.08),  and  the  6  month 
group  had  the  lowest  {M  =  33.64,  SD  =  10.76). 

Support  for  Education:  Significance  was  obtained  for  the  two-way  interaction:  F  (2,  86)  = 
3.41 ,  p  =  .038  (partial  =  .073  which  is  a  medium  effect  size,  meaning  that  7.3%  of  the 
variability  in  the  outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had  very 
similar  scale  means  for  T1  {M  =  11 .74)  and  J2  {M  =  ^^  .95)  followed  by  J3  {M  =  ^  0.86).  Family 
Medicine  obtained  a  relatively  higher  mean  for  T2  (M  =  12.57)  and  lower  for  73  (M  =  ^^  .5)  and 
T1  {M=  10.0). 

A  significant  main  effect  was  also  obtained  for  time:  F  (2,  86)  =  3.94,  p  =  .023  (partial  = 
.084,  which  is  a  medium  effect  size,  meaning  8.4%  of  the  variability  in  the  outcome  is 
attributable  to  differences  between  the  two  levels  of  time).  Collapsing  across  group,  T2  has  the 
highest  mean  {M  =  12.21 ,  SD  =  1 .92)  and  T 1  has  the  lowest  mean  {M  =  ^^ .0,  SD  =  2.45)  though 
any  interpretation  of  a  significant  main  effect  must  be  approached  with  caution  given  the 
significant  result  of  the  first  order  interaction. 

Working  with  Ciinicaiiy  Competent  Peers:  Significance  was  obtained  for  the  two-way 
interaction:  F  (2,  84)  =  3.56,  p  =  .033  (partial  =  .078  which  is  a  medium  effect  size,  meaning 
that  7.8%  of  the  variability  in  the  outcome  is  attributable  to  the  two-way  interaction).  Internal 
Medicine  had  similar  scale  means  for  T1  {M=  12.82)  and  T2  {M  =  12.37)  and  lower  for  T3  {M= 
11.14).  Family  Medicine  obtained  a  relatively  higher  mean  for  T3  {M  =  13.17)  and  lower  for  T 1 
{M=  11.56)  and  T2  {M  =  11.86). 

Ciinicai  Autonomy:  Significance  was  obtained  for  the  two-way  interaction:  F  (2,  85)  =  4.81 , 
p  =  .011  (partial  =  .102  which  is  a  medium  to  large  effect  size,  meaning  that  10.2%  of  the 
variability  in  the  outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had 
similar  scale  means  for  T2  {M  =  80.63)  and  T^  {M=  79.43)  but  quite  a  bit  lower  for  T3  {M  = 

70.0).  The  means  for  FM  were  more  separated,  with  the  highest  mean  at  T2  {M  =  86.57) 
followed  by  T3  (M  =  81 .58)  and  then  T1  {M  =  71 .94). 
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A  significant  main  effect  was  obtained  for  time:  F  (2,  85)  =  4.44,  p  =  .01 5  (partial  rf  =  .095, 
a  medium  to  large  effect  size,  meaning  that  9.5%  of  the  variability  in  the  outcome  is  attributable 
to  differences  between  the  two  time  periods).  Collapsing  across  group,  {M  =  76.36,  SD  = 

1 1 .44)  and  T3  cohorts  have  similar  means  {M  =  77.32,  SD  =  14.72)  compared  to  the  highest 
mean  obtained  by  the  T2  cohort  {M  =  83.15,  SD  =  12.07)  though  any  interpretation  of  a 
significant  main  effect  must  be  approached  with  caution  given  the  significant  result  of  the  first 
order  interaction. 

Control  over  Nursing  Practice:  Significance  was  obtained  for  the  two-way  interaction:  F  (2, 
84)  =  8.41,  p  <  .05  (partial  r)^  =  .167,  a  large  effect  size,  meaning  that  16.7%  of  the  variability  in 
the  outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had  somewhat  similar 
scale  means  for  {M  =  75.59)  and  T2  {M  =  73.89)  but  substantially  lower  (although  n  =  7)  for 
T3  {M  =  51 .0).  Family  Medicine  obtained  the  highest  mean  for  T3  {M  =  82.42)  followed  by  T2  {M 
=  75.64)  and  T1  (M=71.31). 

A  significant  main  effect  was  obtained  for  group:  F  (1 ,  84)  =7.72,  p  =  .007  (partial  rf  =  .084, 
which  is  a  medium  effect  size,  meaning  that  8.4%  of  the  variability  in  the  outcome  is  attributable 
to  differences  between  the  two  levels  of  group).  Collapsing  across  time  FM  has  the  higher 
mean  {M  =  75.93,  SD  =  1 6.74)  than  IM  (M  =  71 .33,  SD  =  1 6.37)  though  any  interpretation  of  a 
significant  main  effect  must  be  approached  with  caution  given  the  significant  result  of  the  first 
order  interaction. 

Perception  that  Staffing  is  Adequate:  Significance  was  obtained  for  the  two-way 
interaction:  F  (2,  86)  =  3.34,  p  =  .04  (partial  rf  =  .074, a  medium  effect  size,  meaning  that  7.4% 
of  the  variability  in  the  outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had 
the  highest  mean  at  T1  {M  =  17.5)  followed  by  T2  {M  =  16.33)  and  then  T3  {M  =  14.86).  Family 
Medicine  obtained  very  similar  means  for  T3  (M  =  1 6.58)  and  T2  (M  =  1 6.36),  then  lowest  for  T 1 
{M  =  14.94).  Neither  of  the  main  effects  were  significant. 

Patient  Centered  Values:  Significance  was  obtained  for  the  two-way  interaction:  F  (2,  85)  = 
8.35,  p  <  .05  (partial  r)^  =  .164,  a  large  effect  size,  meaning  that  16.4%  of  the  variability  in  the 
outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had  the  highest  and 
similar  means  for  T2  {M  =  34.44)  and  {M  =  33.92)  and  lowest  for  T3  {M  =  28.14).  Family 
Medicine  had  the  highest  for  T3  {M  =  37.42)  followed  by  T2  {M  =  33.07),  then  T1  {M  =  29.81 ). 
Neither  main  effects  were  significant. 

Supportive  Nurse  Manager  Relationships:  Significance  was  obtained  for  the  two-way 
interaction:  F  (2,  85)  =  7.8,  p  =  .001  (partial  rf  =  .155,  a  large  effect  size,  meaning  that  15.5%  of 
the  variability  in  the  result  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had  very 
similar  scale  means  for  T^  {M  =  36.7)  and  T2  {M  =  36.53)  and  lower  (although  n  =  7)  for  T3  {M  = 
31 .0)  whereas  FM  obtained  the  highest  mean  for  T3  {M  =  42.67)  followed  by  T2  {M  =  38.14)  and 
T1  (yw=  33.19). 

A  significant  main  effect  was  obtained  for  group:  F  (1 ,  85)  =4.7 1 ,  p  =  .033  (partial  r\^  =  .052, 
which  is  a  medium  effect  size,  meaning  that  5.2%  of  the  variability  in  the  result  is  attributable  to 
differences  between  the  two  levels  of  group).  Collapsing  across  time,  FM  has  the  higher  mean 
{M  =  37.55,  SD  =  7.63)  than  IM  {M  =  35.82,  SD  =  6.78)  though  any  interpretation  of  a  significant 
main  effect  must  be  approached  with  caution  given  the  significant  result  of  the  first  order 
interaction. 
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Professional  Job  Satisfaction:  Significance  was  obtained  for  the  two-way  interaction:  F  (2, 
84)  =  11. 47,  p  <  .05  (partial  rf  =  .214  which  is  a  large  effect  size,  meaning  that  21.4%  of  the 
variability  in  the  outcome  is  attributable  to  the  two-way  interaction).  Internal  Medicine  had  the 
highest  mean  for  T1  {M  =  317.25)  relative  to  T2  {M  =  297.25)  followed  by  T3  {M  =  263.79) 
whereas  FM  obtained  the  highest  mean  for  T3  {M  =  335. 1 5),  then  J2  {M  =  308.93)  and  then  T 1 
{M  =  284.5).  Neither  of  the  main  effects  were  significant. 

A  two-way  MANOVA  was  also  conducted  for  the  8  EOM  subscales  (Figure  1 )  to  ascertain  if 
a  group  x  time  interaction  was  significant  for  the  linear  combination  of  the  EOM  scales.  The 
group  by  time  interaction  was  significant:  Wilk's  A  =  .643,  F  (16,  146)  =  2.26,  p  =  .006  (partial  rf 
=  .198).  This  means  that  the  relationship  of  group  and  the  linear  combination  of  the  8 
outcomes,  in  part,  depends  upon  the  'time'  condition.  Similar  to  the  prior  univariate  analysis,  all 
of  the  two-way  interactions  are  significant  for  the  outcomes,  excepting  RN/MD  Relationships 
and  for  the  multivariate  model  Staff  Perception  that  Staffing  is  Adequate  is  now  not  significant  (p 
=  .052).  The  only  significant  main  effect  for  group  was  found  for  Control  over  Nursing  Practice. 
Significant  main  effects  for  the  time  condition  were  found  for  RN/MD  Relationships,  Support  for 
Education,  and  Clinical  Autonomy. 


FIGURE  1:  MANOVA  results  for  the  EOM 
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Qualitative  data. 

One  overarching  theme,  satisfaction,  supported  categories  from  both  clinics  (table  3). 

Categories  were  similar  in  both  clinics  except  for  communication  and  quality  of  care,  which 
emerged  from  IM  only.  Comments  in  either  category  were  not  more  positive  over  time:  “a  lack  of 
equipment  and  support  staff  has  made  it  very  difficult  to  provide  the  quality  care  I  feel  we  should 
be  able  to  provide”  (IM  T-2;  ‘We  need  to  have  an  open  lines  of  communication  with  one 
another”  (IM  T-3). 

Access  was  an  issue:  “Appointments  are  not  given  adequate  time”  (FM,  T-3);  “No  appointments 
available  have  to  find  way  to  have  see  anyway  despite  the  level  of  acuity  for  fear  of  negative 
ICE  Comment”  (IM  T-2).  Staff  in  both  clinics  felt  that  while  they  had  some  autonomy,  many 
decisions  were  still  being  made  without  their  input:  “We  do  have  input  on  the  unit  practice 
council  and  nurse  practice,  but  I  feel  it  is  chosen  by  higher  ups  what  these  councils  can  provide 
input  on  and  what  they  cannot”  (IM,  T-1 );  “Patient  care  is  a  top  priority  here  at  IMC,  but  decision 
is  not  based  on  agreed  consensus  by  everyone”  (IM  T-3).  “If  you  want  people  to  act  professional 
treat  them  as  professional  and  not  monkeys  doing  tasks.”  (FM  T-2). 

Staff  struggled  with  professional  practice  yet  remained  focused  on  teamwork:  “This  place  I 
believe  strongly  has  turned  for  the  worse  in  which  the  RN’s  are  nowhere  near  practicing  at  the 
top  of  our  scope.  ”(IM  T-3).  “Over  50%  of  what  the  nurses  do  could  be  done  by  MSAs  and  NAs” 
(IM  T-2).  “Difficult  to  keep  staff  because  everyone  is  not  on  the  same  page,  but  everyone  tries  to 
work  as  a  team.”  (IM  T-1).  Practice  issues  were  not  limited  to  nurses:  “Medics  need  to  be  able  to 
do  what  our  training  has  taught  us  not  what  it  is  thought  we  should  do”  (FM  T-2). 

Change  was  a  concern  in  both  clinics:  “Too  many  changes  in  past  six  months”  (FM,  T-3); 
“Sometimes  too  many  changes  at  once.  Changes  are  good,  however  not  when  it  confuses 
staff.”  (IM,  T-1).  Resources  to  deal  with  changes  were  felt  to  be  lacking:  “With  all  the  additional 
tasks  and  duties  given  to  us,  we  need  to  be  provided  with  adequate  resources  (both  staff  and 
equipment)”  (FM,  T-3). 

Despite  these  issues,  some  staff  remained  satisfied  with  their  job.’  “I  truly  love  my  job  but  if 
only  we  can  have  the  equipment  and  good  communication  system  I’ll  be  even  more  satisfied” 
(IM  T-2);  “Overall,  people  in  this  clinic  work  well  together  (for  the  most  part)”  (FM  T-3).  A 
complete  listing  of  all  categories  and  comments  are  in  appendix  B. 

Patient  Satisfaction 

PSQ-18\  For  the  mean  averaged  scales  for  the  PSQ-18  (scores  range  from  a  low  of  1  to  a 
high  of  5)  the  highest  mean  was  obtained  for  the  Interpersonal  Manner  scale  (M  =  4. 1 3,  SD  = 
.80)  followed  by  Communication  {M  =  4.10,  SD  =  .78)  and  the  lowest  mean  was  obtained  for  the 
Accessibility  and  Convenience  scale  {M  =  3.68,  SD  =  .83).  When  conducting  the  two-way 
ANOVA  (group  by  time)  a  significant  two-way  interaction  was  found  only  for  Access  and 
Convenience:  higher  means  were  obtained  at  T-3  for  both  IM  {M  =  3.65,  SD  =  .84)  and  FM  (M  = 
3.92,  SD  =  .84)  compared  to  T-1  or  T-2. 

Significant  main  effects  for  group  were  found  with  higher  means  in  FM  than  IM  for 
Technical  Quality  (FM:  M  =  3.99,  SD  =  .78;  IM:  M  =  3.87,  SD  =  .72),  Interpersonal  Manner  (FM: 
M  =  4.2,  SD  =  .79;  IM:  M  =  4.07,  SD  =  .80),  Time  Spent  Dr./Provider  (FM:  M  =  3.89,  SD  =  .95; 

IM:  M  =  3.73,  SD  =  .90),  and  Access  and  Convenience  (FM:  M  =  3.78,  SD  =  .86;  IM:  M  =  3.59, 
SD  =  .80),  though  interpretation  of  the  significant  main  effect  is  subordinated  to  the  higher  order 
significant  two-way  interaction. 
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For  time,  Communication  and  Access  and  Convenience  have  significant  between-time 
differences,  with  higher  means  for  both  scales  obtained  at  T-3  (F  (2,  1189)  =  5.45,  p  =  .004, 
partial  ri^  =  .009  and  F(2,  1185)  =  3.83,  p  =  .022,  partial  ri^  =  .006  respectively).  The  very  small 
effect  sizes  mean  that  .9%  and  .6%  of  the  variability  in  the  outcome  is  attributable  to  differences 
between  the  two  time  periods.  Collapsing  across  group,  for  communication,  the  T-3  group  has  a 
higher  mean  {M  =  4.18,  SD  =  .75)  than  T-1  {M  =  4.13,  SD  =  .77)  and  T-2  {M  =  4.00,  SD  =  .82). 
For  access  and  convenience,  the  T-3  cohort  has  the  highest  mean  {M  =  3.79,  SD  =  .85)  than  T- 
^{M  =  3.69,  SD  =  .83)  and  J-2  {M  =  3.57,  SD  =  .81)  though  interpretation  of  the  significant  main 
effect  is  subordinated  to  the  higher  order  significant  two-way  interaction. 

For  simultaneous  analysis  of  all  6  PSQ  scales,  the  two-way  MANOVA  test  of  the  group  x 
time  interaction  was  not  significant:  Wilk's  A  =  .991 ,  p  =  .55  (partial  rf  =  .005).  The  main  effect 
for  group  was  significant:  Wilk's  A  =  .982,  p  =  .001  (partial  ri^  =  .01 8)  and  as  well  for  the  main 
effect  for  time:  Wilk's  A  =  .976,  p  =  .004  (partial  ri^  =  .012).  Examination  of  the  univariate  results 
are  such  that  main  effects  for  group  are  significant  for  all  of  the  individual  outcomes  excepting 
General  Satisfaction  (p  =  .065)  with  Family  Medicine  obtaining  higher  mean  values  than  Internal 
Medicine. 

APLSS:  Items  analyzed  for  satisfaction  question  were:  Overall  Satisfaction,  Overall 
Satisfaction  with  PCM  (reported  as  one  score),  staff  courtesy/helpfulness,  overall  visit 
satisfaction,  and  satisfaction  with  healthcare  overall.  Family  Medicine  was  lower  at  T3  than  T1 
or  T2  for  all  subscales.  Internal  Medicine  was  higher  at  T3  than  T1  or  T2  for  overall  satisfaction, 
satisfaction  with  healthcare  overall,  time  between  scheduling  and  visit,  and  wait  time  in  clinic. 
Specifically,  for  the  2  (group)  x  3  (time)  binary  logistic  regression,  a  significant  interaction  was 
found  for  the  following: 

Overall  Visit  Satisfaction:  there  was  a  slightly  higher  percent  for  the  FM  group  at  T2 
(94.8%)  but  an  appreciably  higher  percentage  (95.1%)  for  IM  when  compared  to  FM  (89.4%)  for 
T3  (figure  2). 

FIGURE  2:  APLSS  Subscale  Overall  Visit  Satisfaction 
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FIGURE  3:  APLSS  Subscale  Time  Between  Schedule  and  Visit 


FIGURE  4\  APLSS  Subscale  Wait  Time  in  Clinic 


ICE:  Items  analyzed  from  this  questionnaire  were:  overall  satisfaction,  staff  attitude, 
timeliness,  quality  of  care,  access  to  medical  care,  and  referral  to  specialty  care.  Sample  size 
was  significantly  different  at  T3  for  IM  than  for  FM  possibly  affecting  outcomes.  A  significant 
interaction  was  found  for  all  subscales  with  FM  higher  than  IM  at  T3  (figures  5-7).  FM  was 
higher  at  T3  than  at  baseline  for  all  scales  except  overall  satisfaction  and  referral  to  specialty 
care  where  mean  was  essentially  the  same  across  time.  Specifically,  for  the  2  (group)  x  3  (time) 
binary  logistic  regression  for  the  ICE  subscales,  a  significant  interaction  was  found  for  all  of  the 
outcomes,  though  interpretation  should  be  approached  with  caution  given  the  very  large 
confidence  intervals  and  also  the  small  sample  size  for  the  last  two  time  periods  for  the  IM 
group:  (1)  Overall  Satisfaction:  the  FM  group  has  the  same  percentage  (99%)  across  all  three 
time  periods  whereas  the  IM  group  sees  an  appreciable  decrease  in  overall  satisfaction  for  T2 
and  T3(88%  and  42%);  (2)  Staff  Attitude:  the  IM  group  has  a  higher  percentage  than  FM  forT1 
(96%  vs.  80%)  and  T2  (87%  vs.  82%)  whereas  the  FM  T3  cohort  has  a  higher  endorsement  of 
staff  attitude  than  IM  (85%  vs.  36%); 

FIGURE  5:  ICE,  Overall  Satisfaction  and  Staff  Attitude 
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(3)  Timeliness:  the  IM  group  has  a  higher  percentage  than  FM  at  T1  (93%  vs.  77%)  and  T2 
(85%  vs. 78%)  whereas  the  FM  T3  cohort  has  a  higher  endorsement  of  timeliness  than  IM  (83% 
vs.  50%);  (4)  Quality  of  Care:  the  IM  group  has  a  higher  percentage  than  FM  for  T 1  (95%  vs. 
84%)  and  T2  (91%  vs. 86%)  whereas  the  FM  T3  cohort  has  a  higher  endorsement  of  Quality  of 
Care  than  IM  (88%  vs.  64%);  (5)  Access  to  Care:  the  IM  group  has  a  higher  percentage  than 
FM  for  T 1  (90%  vs.  76%)  whereas  the  FM  T3  cohort  has  a  higher  endorsement  of  Access  to 
Care  than  IM  (82%  vs.  56%).  Both  groups  have  79%  endorsement  for  T2;  (6)  Referral  to 
Specialty:  the  IM  group  has  a  higher  percentage  than  FM  for  T 1  (89%  vs.  81  %)  and  T2  (86% 
vs.  79%)  whereas  the  FM  T3  cohort  has  a  higher  endorsement  of  Referral  to  Specialty  care 
than  IM  (82%  vs.  33%). 

FIGURE  6:  ICE  Questions  Referral  to  Specialty  Care  and  Quality  of  Care 


FIGURE  7:  ICE  Questions  Timeliness  and  Access  to  care 


Qualitative  data 

Both  patient  and  staff  data  categorized  as  nonclinical  referred  to  issues  outside  of  the 
immediate  effects  normally  associated  with  outpatient  professional  practice:  parking,  inpatient 
care,  pharmacy  services,  emergency  services,  and  specialty  care  such  as  podiatry  tended  to  be 
negative. 

For  patients,  one  overarching  theme,  quality  of  care,  supported  seven  categories:  access, 
communication,  continuity,  lack  of  resources,  non-clinical,  quality  of  care,  and  satisfaction  with 
care  (table  3).  Categories  were  similar  in  both  clinics  across  all  time  periods  except  for  lack  of 
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resources,  which  emerged  from  FM  at  T1  only  “Please  enable  our  providers  what  they  need  to 
continue  great  [sic]  care  to  us”.  Positive  and  negative  comments  were  obtained  in  both  clinics 
for  all  time  periods,  with  negative  comments  somewhat  more  predominate  across  clinics  and 
time  periods.  Access  issues  were  slanted  towards  specialty  care:  “Difficult  or  too  long  to  get 
referrals  for  specialists”  FM,  T1 ;  but  also  referenced  availability  “Hard  to  get  appointments  with 
PCM  without  waiting  2  weeks”  FM  T3,  “Its  hard  when  the  clinic  Is  so  busy  and  no  sick  call  on 
weekends”  IM  T3;  and  appointment  system  “The  appointment  line  Is  horrible”  IM  T1; 
Suggestions  were  made  as  to  how  to  improve  access:  “more  providers  are  needed  to 
accommodate  the  growing  needs  of  the  military  “  FM  T1 ;  “Give  providers  more  time  to  see 
patlentsi”  \M  T2,  “we  need  an  acute  care  clinic”  IM  T3.  Improvements  were  also  noted:  “The 
new  appointment  system  works  efficiently  and  staff  very  helpfuT  IM  T1 ,  “/  like  the  after  hours 
clinic  service,  great  to  accommodate  those  of  us  with  busy  schedules”  IM  T2,  “I  am  accustomed 
to  waiting  20-30  minutes  past  my  appointment  time  for  care.  This  was  not  the  case  for  my  first 
visit  to  this  clinic”  FM  T2. 

Communication  addressed  system  issues,  providers,  and  support  staff:  “One  system 
hospital  Information  one  way  and  the  other  has  patient  Information  another  way.  No  one  wants 
to  take  the  time  to  ensure  that  all  the  Information  Is  correct  FM  T1 ,  “Sometimes  tests  results 
are  explained;  usually  not  unless  I  ask’  FM  T2,  “They  should  kindly  Inform  you  they  are  behind 
so  you  know  what  to  expect  IM  T2,  “When  messages  are  left  for  doctors  or  nurses  98%  there  Is 
not  return  call  to  confirm”  IM  T3. 

Continuity  addressed  providers,  nurses,  and  the  military  system:  “why  assign  patients  to  a 
provider  they  never  see?”  FM  T1 ,  “My  PCM  has  changed  frequently,  but  I’ve  grown  to  expect 
that  with  the  military”  FM  T3,  “Always  get  an  appointment  with  someone  on  team”  FM  T3,  “I 
would  like  to  have  the  same  nurse  every  appointment  If  possible”  FM  T3,  “Td  really  like  It  If  I 
could  see  the  same  person  all  the  time”  IM  T1 ,  “Every  time  I  visit  -  which  Is  Infrequent  my 
primary  care  team  changes.  Not  a  problem  -  just  stopped  trying  to  keep  track’  IM  T3. 

Quality  of  care  comments  were  typically  more  provider  focused:  “My  personal  opinion  Is  the 
utilization  of  Nurse  Practitioners  at  TMC  Is  a  point  of  genius!!”  FM  T1 ,  “/  feel  I  asked  all  the 
question.  He  was  not  good  explaining  to  me  at  alf’  FM  T2,  “Concerned  that  doctors  are  more 
concerned  for  patients  or  deployable  than  finding  or  fixing  patients  medical  Issues”  IM  T1 , 
“These  questions  depend  on  the  Individual  doctor  as  of  now  Dr.  —  Is  great’  IM  T3,  “Ever  since 
being  provided  with  the  “white  teams”  contact  phone  numbers,  I  have  had  quick  response  to  all 
my  Inquiries”  IM  T3. 

Comments  regarding  satisfaction  were  more  general:  “I  am  here  on  R  &  R  as  a  medic  In  Iraq 
It  Is  nice  to  know  my  family  Is  medically  sound’  FM  T1 ,  “Overall  I  am  very  pleased  with  the 
service  provided  by  this  facility’  FM  T2,  “They  need  to  spend  more  time  reviewing  patient 
records  and  getting  to  know  this  patient,  this  requires  time  which  means  providers  cannot  be 
overbooked’  FM  T3,  “I’ve  seen  lots  of  Improvement  In  medical  services/care  provided  to  me”  IM 
T 1 ,  “Satisfied  and  thank  you”  IM  T2,  “I  love  the  treatment  I  get,  but  Instead  of  worrying  about 
what  good  or  bad  feelings  people  have  about  Tripler,  we  all  would  better  served  If  people  health 
problems  were  cured  completely’  IM  T3.  A  complete  listing  of  all  categories  and  comments  are 
in  appendix  B. 

In  summary,  FM  staff  satisfaction  improved  over  time  and  over  half  (7  of  12)  of  the  patient 
satisfaction  measures  maintained  or  met  target  over  time.  Internal  medicine  clinic  staff 
satisfaction  declined  over  time,  while  more  than  half  (8  of  12)  of  patient  satisfaction  metrics 
maintained  or  met  target  over  time.  Qualitative  data  provided  insight  with  regards  to  specific 
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metrics  that  did  or  did  not  improve.  Staff  comments  addressed  overall  satisfaction  while  patient 
comments  targeted  quality  of  care. 

Aim  two 

Evaluate  the  effects  of  implementation  of  a  PPM  on  baseline  measures  of  patient 
satisfaction,  indicators  of  quality  and  access  to  care  measures  in  ambulatory  care  clinics. 

Patient  Satisfaction.  Please  refer  to  aim  one. 

Quality  care.  For  the  2  (group)  x  3  (time)  binary  logistic  regression  for  the  HEDIS  outcomes, 
no  significant  group  x  time  interaction  was  found  for  any  of  the  outcomes.  However,  there  were 
significant  effects  for  group  and/or  time  as  follows:  (1 )  Mammogram:  the  IM  group  had  a  higher 
overall  percentage  of  eligible  patients  with  mammogram  for  each  of  the  time  cohorts  when 
compared  to  FM  and  for  both  of  the  medical  groups  there  was  a  decrease  in  percent  of  eligible 
patients  with  mammogram  for  each  of  the  successive  time  cohorts;  (2)  Long-Term  Asthma 
Medication:  the  FM  group  has  a  higher  overall  percent  of  eligible  patients  with  long  term  asthma 
medication  for  the  last  two  time  cohorts  when  compared  to  IM;  (3)  A1 C  T esting:  the  IM  group 
had  a  higher  overall  percent  of  eligible  patients  with  A1C  testing  for  all  of  the  time  cohorts  when 
compared  to  FM;  (4)  A1C  less  than  or  equal  to  9:  the  IM  group  had  a  higher  overall  percent  of 
eligible  patients  with  A1C  less  than  or  equal  to  9  for  all  of  the  time  cohorts  when  compared  to 
FM;  (5)  LDL  less  than  or  equal  to  100:  the  IM  group  had  a  higher  overall  percent  of  eligible  with 
LDL  less  than  or  equal  tolOO  for  all  of  the  time  cohorts  when  compared  to  FM;  (6)  Cervical 
Screening:  the  FM  group  had  a  higher  overall  eligibility  with  Cervical  Screening  for  all  of  the  time 
cohorts  when  compared  to  IM;  (7)  Colorectal  Screening:  the  FM  group  has  a  higher  overall 
eligible  with  Colorectal  Screening  for  the  first  two  time  cohorts,  though  IM  has  a  higher  mean 
value  for  the  last  time  cohort;  (8)  Chlamydia  Screening:  the  IM  group  had  a  higher  overall 
eligible  with  Chlamydia  Screening  for  all  of  the  time  cohorts  when  compared  to  FM;  (9)  Adult 
Pneumococcal  Immunization:  the  FM  group  had  a  higher  overall  eligible  with  Adult 
Pneumococcal  Immunization  for  each  of  the  time  cohorts  when  compared  to  IM.  The  only 
measure  to  improve  over  time  was  percent  of  eligible  patients  with  long  term  asthma  medication 
in  FM. 

Goals  for  HEDIS  measures  change  annually  based  on  the  population  or  N  of  participating 
hospitals  nationwide  reporting  data  year  to  year.  Clinics  are  rated  green  (optimal),  amber,  or 
red  if  they  achieve  at  least  90%,  75%,  or  50%  of  the  goal  respectively.  Both  clinics  maintained 
or  achieved  green  by  T3  for  chlamydia  and  cervical  screening.  Family  Medicine  also  maintained 
or  achieved  green  status  by  T3  for  long-term  asthma  medication  and  IM  for  colorectal  screening 
(table  4). 

Access  to  care.  Specifically,  for  the  2  (group)  x  3  (time)  binary  logistic  regression,  a 
significant  interaction  was  found  for:  (1)  Time  between  scheduling  and  visit:  the  clinics  have 
equivalent  percentages  for  the  T 1  (85.6%)  whereas  the  FM  T2  cohort  has  a  higher  endorsement 
(81.1%  vs.  78.1%)  compared  to  the  IM  T3  cohort  which  obtains  a  higher  endorsement  (87.6% 
vs.  79.9%);  (2)  Wait  Time  in  Clinic:  the  IM  group  has  higher  endorsement  for  the  T 1  cohort 
(82.6%  vs.  81.2%)  and  T3  (88.5%  vs.  76%)  whereas  the  FM  T2  cohort  has  a  higher 
endorsement  (83.2%  vs.  79.1%). 

Continuity.  For  PCM  by  name  both  FM  (T1  =50.33%,  T2=58.33%,  T3=62.67%)  and  IM 
(T1  =54.67%,  T2=55%,  T3=62.67%)  moved  from  red  or  amber  status  to  green  (>60%)  by  T3. 

For  PCM  by  team  FM  (T1=91.67%,  T2=94.33%,  T3=92%)  and  IM  (T1=91%,  T2=89.3%, 
T3=93.67%)  maintained  green  status  (>85%). 
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In  summary,  Access  and  continuity  measures  followed  similar  patterns  as  the  satisfaction 
items.  Specifically,  about  half  of  the  metrics  maintained  or  reached  target  goals  by  T3  in  FM  (5 
of  10)  and  IM  (5  of  10).  Quality  of  care  metrics  (n=9)  were  somewhat  less  robust  in  that  few 
metrics  in  either  clinic  maintained  or  reached  target  over  time  (FM  n=2,  IM  n=3).  Several  metrics 
worsened  over  time  (FM  n=3,  IM  n=2).  as  noted  in  Table  4. 

Table  4 

Status  of  HEDIS  and  continuity  goais  by  ciinic  and  time 


Metric 


Family  Medicine 
T1  T2  T3 


Internal  Medicine 
T1  T2  T3 


Mammogram 

Long-term  asthma  medication 

Ale  testing 

Ale  <9 

LDL<  100 

Cervical  screening 

Colorectal  screening 

Chlamydia  screening 

Adult  pneumovax 

PCM  by  name 

PCM  by  team 


amber 

red 

red 

green 

green 

green 

amber 

amber 

amber 

red 

red 

red 

amber 

amber 

red 

green 

green 

green 

green 

green 

amber 

red 

red 

red 

amber 

amber 

N/A 

red 

amber 

green 

green 

green 

green 

green 

green 

amber 

green 

green 

amber 

amber 

amber 

amber 

amber 

amber 

amber 

green 

green 

green 

green 

green 

green 

green 

green 

green 

red 

red 

red 

amber 

amber 

N/A 

amber 

amber 

green 

green 

green 

green 

Aim  three 

Evaluate  the  relationship  between  staff  and  patient  satisfaction  and  quality  of  care 
outcomes  following  the  implementation  of  the  PPM. 

Quantitative  data.  Thirty-two  items  (staff  satisfaction  n=1,  patient  satisfaction  n=13,  quality 
of  care  n=9,  and  access/continuity  n=10)  were  assessed.  As  discussed  in  aim  one  and  two,  the 
following  measures  improved  over  time  and  met  target  goals  in  FM:  staff  satisfaction  (EOM-II®), 
5  of  12  patient  satisfaction  items  (PSQ-18,  ICE,  APLSS),  1  of  9  quality  of  care  items  (HEDIS), 
and  5  of  10  access/continuity  items.  For  IM,  measures  that  improved  over  time  and  met  goals 
included  6  of  12  patient  satisfaction  items  and  4  of  10  access/continuity  items  (table  5).  In  FM  3 
of  12  patient  satisfaction  items  met  target  levels  at  T3  but  were  lower  than  T 1  or  T2.  Measures 
with  lower  scores  at  T3  that  still  met  target  goals  in  IM  included  2  of  12  patient  satisfaction 
items,  3  of  9  quality  of  care  items,  and  3  of  10  access/continuity  items.  Some  measures  in  both 
clinics  met  target  goals  at  T1  or  T2  but  not  at  time  T3  or  improved  over  time  but  did  not  achieve 
the  target.  Overall,  each  clinic  met  or  maintained  target  goals  at  T3  for  about  half  (FM  n=16;  IM 
n=17)  of  the  measures.  Qualitative  data  provided  more  insight  regarding  staff  and  patient 
perceptions  over  the  course  of  this  project  (appendix  B).  Points  of  divergence,  convergence, 
and  complementarity  with  quantitative  data  (table  6)  were  as  follows: 

Divergence.  The  EOM-II®  scores  indicated  FM  had  a  very  healthy  work  environment  by  T3 
yet  staff  comments  identified  issues  such  as  a  lack  of  resources,  time,  and  leadership  support 
as  affecting  morale,  burnout,  and  staff  retention:  Too  much  is  expected  from  too  iittie  (FM,  T-1); 
Burnout  for  some  peopie  is  a  reai  issue  (FM  T-3);  The  femaie  charge  nurse  is  a  buiiy  and  picks 
favorites  (FM,  T-3);  Internal  Medicine  scores  declined  from  a  healthy  work  environment  at  T 1 
and  T2  to  a  work  environment  needing  improvement  at  T3:  The  moraie  was  tow  at  one  point  but 
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/  think  after  all  of  us  voiced  our  opinions  recently,  the  morale  have  gotten  a  little  better.  Thank 
you  (IM  T-3) 

Patient  satisfaction  was  higher  in  both  clinics  at  T3,  yet  not  all  patient  experiences  were 
positive:  there  are  some  experiences  that  were  truly  grotesque!  (FM  T2);  Communication  about 
rescheduling  is  very  poor.  When  messages  are  left  for  doctors  or  nurses  98%  there  is  not  return 
call  to  confirm  about  refills  or  appointments.  (IM  T3). 

Similarly,  access/continuity  measures  met  target,  but  remained  problematic  for  some:  Hard 
to  get  appointment  with  PCM  in  less  than  2  weeks  time  (FM  T3);  /  think  we  need  an  acute  care 
clinic.  Keep  the  ER  from  getting  backed  up,  but  it’s  hard  when  the  clinic  is  so  busy  and  no  sick 
call  on  weekends  (IM  T3) 

Few  quality  of  care  items  met  target  by  T3,  yet  patient  comments  were  more  positive  than 
negative:  have  never  had  better  care  or  need  more  efficient  health  care  providers  (FM  T2);  NP 
XX  and  his  nurse  took  time  to  listen  to  me  and  understood  my  frustrations  (IM  T3). 

Convergence:  As  previously  noted,  about  half  of  the  metrics  met  target  goals  by  T3.  Both 
staff  and  patients  indicated  progress  was  being  made,  but  professional  practice, 
access/continuity  and  quality  of  care  remained  opportunities  for  improvement:  We  have  a  lot  of 
independence  and  autonomy  in  our  roles,  but  sometimes  decisions  are  made  from  higher  up 
without  our  input  (EOM-II®  IM  T1 );  Appointments  are  not  given  adequate  time  (EOM-II®  FM,  T- 
3);  Have  no  say  whatsoever  in  nursing  practice  here  (EOM-II®  IM  T-3);  Keep  up  the  great  work 
that  you  all  are  doing,  I  give  all  of  you  a  big  A+  (PSQ  IM  T1 );  /  really  dislike  the  fact  my  primary 
care  doctor  can  be  switched  without  little  to  no  input  from  me.  (PSQ  FM  T3);  Thank  you  folks  for 
always  making  great  improvements  to  service  all  patients  (PSQ  IM  T3);  We  have  always 
received  the  very  kindest,  most  thorough,  and  attentive  care  from  everyone  at  Tripler  (PSQ  FM 
T2);  Thanks  for  being  great  Doctors  and  nurses  (PSQ  FM  T3). 

Complementarity.  Qualitative  data  provided  a  bit  more  insight  regarding  satisfaction, 
access/continuity  metrics,  and  quality  of  care  from  staff  and  patient  perspectives:  A  lot  of  waste 
(equipment,  time  and  energy  on  new  ideas  because  of  “command”  demands)  (EOM-II®  IM  T-1); 
Decisions  that  directly  affect  our  ability  to  do  our  jobs  in  terms  of  logistics,  ergonomics,  and 
equipment  availability  and  selection  are  made  with  little  or  no  input  from  staff  (EOM-II®  IM  T-3); 
Overall,  people  in  this  clinic  work  well  together  (for  the  most  part)  (EOM-II®  FM  T-3);  I  wish  I 
didn’t  have  to  see  my  PCM  for  OBGYN  issues.  Distinctly  uncomfortable  with  that  (PSQ  FM  T2); 
Never  is  appointment  on  time  -  always  waiting  (PSQ  IM  T2);  Always  get  an  appointment  with 
someone  on  team  (PSQ  FM  T3);  Dr.  XX  staff  are  very  professional  and  courteous  (PSQ  IM  T3). 

In  summary,  over  half  of  the  metrics  tracked  in  this  project  improved  over  time  or 
achieved/maintained  target  goals  by  T3.  While  these  results  were  not  as  robust  as  hoped  for, 
both  clinics  made  progress  overall  in  the  right  direction.  Qualitative  data  was  similarly  equivocal 
in  that  data  were  bidirectional.  Neither  positive  nor  negative  comments  prevailed  and  some 
insight  regarding  quantitative  data  results  emerged.  Staff  satisfaction  in  both  clinics  centered  on 
lack  of  resources,  change  chaos,  communication,  and  practicing  to  the  full  extent  of  their 
education  and  training.  Patient  satisfaction  converged  around  elements  of  quality  care:  access, 
communication,  continuity,  and  technical  quality.  Both  staff  and  patients  were  cognizant  of 
progress  and  identified  specific  opportunities  for  improvement. 
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Table  5 

Summary  of  metrics  meeting  target  goais  by  time  and  ciinic 


Metric 

Family  Medicine 

Internal  Medicine 

By  Time 

* 

- n - 

By  Clinic 

T1 

12 

T3 

T1 

12 

T3 

T1  12 

T3 

FM  IM 

Satisfaction 

X 

X 

X 

X 

EOM  II 

PSQ 

X 

X 

X 

General  satisfaction 

Technical  quality 

X 

X 

X 

X 

Communication 

X 

X 

X 

Interpersonal  manner 

X 

X 

X 

ICE 

Overall  satisfaction 

X 

X 

X 

Staff  attitude 

X 

X 

X 

X 

Quality  of  care 

X 

X 

APLSS 

X 

X 

X 

X 

X 

Overall  satisfaction 

Satisfaction  w  PCM 

X 

X 

X 

Courtesy/respect 

X 

X 

X 

X 

X 

X 

X 

Visit  satisfaction 

X 

X 

Satisfaction  w  healthcare 

X 

X 

X 

Access/  Continuity 

PSQ 

X 

X 

X 

X 

X 

Access/convenience 
Time  spent  with  PCM 

X 

X 

X 

ICE 

X 

X 

Access  to  care 

Timeliness 

X 

X 

X 

Referral  to  specialty 

X 

X 

X 

APLSS 

Time  to  scheduled  visit 
See  PCM  when  needed 

Wait  time  in  clinic 

X 

X 

Other 

X 

X 

X 

X 

X 

X 

X 

PCM  by  name 

PCM  by  team 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Quality  of  care 

X 

X 

X 

Mammogram 

Long  term  asthma 

X 

X 

X 

X 

X 

X  X 

X 

X 

medication 

A1c  testing 

X 

A1c<9 

X 

LDL  < 100 

X 

X 

X 

X 

Cervical  screening 

X 

X 

X 

X 

X 

X 

X 

X 

Colorectal  screening 

X 

X 

X 

X 

X 

X 

Chlamydia  screening 

X 

Adult  pneumovax 

X 

Note:  ‘metric  met  target  during  time  period  regardless  of  clinic;  **  clinic  with  higher  mean  score  coliapsed  across 
time  regardless  whether  or  not  target  was  achieved. 
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Table  6 


Convergence,  divergence,  and  complementarity 


Metric 

Satisfaction 


Family  Medicine 
T-1  T-2  T-3 


EOM-II® 


PSQ  X 

General  satisfaction 

Technical  quality  X 

Communication  X 

Interpersonal  manner 

ICE  ^ 

Overall  satisfaction 

Quality  of  care  X*  X* 

Staff  attitude  X  X 

APLSS  w**  w** 


Overall  satisfaction 
Satisfaction  w/  PCM  X** 

Courtesy  and  respect  X**  X**  X** 

Visit  satisfaction 
Satisfaction  w/  healthcare 
Access/continuitv 

PSQ  X 

Access  and  convenience 


Internal  Medicine 
T-1  T-2  T-3 


X  X 


X  X 

X  X 

X 

X  X 


X 

X 

X 


** 

** 

** 


X** 

X**  X 

X  X 

X  X 


Exemplar 

If  you  want  people  to  act  professional  treat  them  as 
professional  and  not  monkeys  doing  tasks  (FM  T-2) 
Overall,  people  in  this  clinic  work  well  together  (for  the  most 
part)  (FM  T-3). 

Sometimes  too  many  changes  at  once.  Changes  are  good, 
however  not  when  it  confuses  staff.  (IM,  T-1) 
Communication  is  another  area  that  could  use  improvement. 
We  often  find  out  about  new  practices  in  other  areas  of 
the  clinic  as  they  are  happening  (IM,  T-1) 

This  place  I  believe  strongly  has  turned  for  the  worse  .  .  .the 
RN’s  are  nowhere  near  practicing  at  the  top  of  our  scope. 
We’re  full  time  telephone  operators  literally  (IM  T-3) 

Lady  at  front  desk  for  sick  call  Family  Practice  needs  to  be 
more  customer/soldier  friendly!  (PSQ-FM  T 1 ) 

I  think  with  as  busy  as  you  are  I  have  seen  improvements 
(PSQ-FM  T2) 

Everyone  is  doing  great... Give  them  a  raise!!  (PSQ-FM  T3) 
Thanks  I  am  very  happy  . .  .  from  the  check  in  desk  to  the 
PCM  (PSQ-FM  T3) 
staff  very  helpful  (PSQ-IM  T 1 ) 

Great  Service  (PSQ-IM  T2) 

Medical  staff  are  very  professional  /  courteous  (PSQ-IM  T3) 
Very  satisfied  with  the  service  I  received.  Improvements  have 
been  noted  the  last  1-2  years  (PSQ-IM  T3) 

Dr.  XX  is  great.  She  listens  and  is  very  detailed  (PSQ-FM  T2) 
MAJ  XX  is  very  thorough  and  explains  in  detail  my  needs  and 
has  even  found  more  ailments!  (PSQ-FM  T3) 
Appointments  not  given  adequate  time  (EQM-II®  FM  T-3) 

No  appointments  available  have  to  find  way  to  have  see 
anyway  despite  the  level  of  acuity  for  fear  of  negative  ICE 
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Time  spent  with  PCM 
ICE 

Access  to  care 
Timeliness 
Referral  to  specialty 

APLSS 

Time  to  scheduled  visit 
See  PCM  when  needed 
Wait  time  in  clinic 


X 


X 


Other 


PCM  by  name 
PCM  by  team 


Quality  of  care 

Mammogram 
Long  term  asthma 
medication 
A1c  testing 
A1c<9 
LDL  < 100 
Cervical  screening 
Colorectal  screening 
Chlamydia  screening 
Adult  pneumovax 


X 


X 


X** 

X** 


X 

X 


X**  X 

X  X 


X 


X* 


X** 

X** 

X** 


X 


X* 


X  X* 


X** 

X** 

X** 


X* 


X* 


X 


X* 


X* 


X** 

X** 

X** 


X* 


X** 

X 


X 


X** 

X** 

X** 


Comment  (EOM-II®  IM  T-2) 

Same  day  appointments  are  hard  to  get  (PSQ-FM  T-1) 

The  appointment  line  is  always  busy  and  the  call  back  option 
have  not  worked  for  me  yet  (PSQ-FM  T-2) 

Very  difficult  to  see  my  PCM!  (PSQ-FM  T 1 ) 

Have  had  3  different  PCM’s  in  last  year  (FM  T2) 

They  always  look  like  they  actually  love  their  work  (PSQ-FM 
T3) 

Love  the  extended  clinic  hours.  4-17  (PSQ-IM  T-1) 

Ever  since  being  provided  with  the  “white  teams”  contact 
phone  numbers,  I  have  had  quick  response  to  all  my 
inquiries  (PSQ-IM  T2) 

Every  time  I  visit  -  which  is  infrequent  my  primary  care  team 
changes.  (PSQ-IM  T3) 

Too  many  changes  in  past  six  months  (EQM-II®  FM,  T-3) 
Everyone  on  staff  here  is  very  compassionate  toward  patients 
and  other  staff  (EQM-II®  IM  T-1) 

In  some  cases  a  lack  of  equipment  and  support  staff  has 
made  it  very  difficult  to  provide  the  quality  care  I  feel  we 
should  be  able  to  provide  (EQM-II®  IM  T2) 
some  of  the  young  doctors  need  better  training  (PSQ-IM  T1) 
The  nurse  taking  my  vitals  was  courteous  and  knowledgeable 
(PSQ-  IM  T2) 

My  health  care  team  is  phenomenal!  (PSQ-IM  T3) 

Nurse  who  gave  me  a  flu  shot  today  did  not  tell  me  her  name 
or  did  not  ask  for  my  identity  (PSQ-FM  T 1 ) 

I  am  a  retired  RN  of  33  years  -  I’ve  lived  all  over  the  United 
States  and  have  never  had  better  care  or  need  more 
efficient  health  care  providers  (PSQ-  FM  T2) 

Very  friendly  nurse  family  oriented  (PSQ-FM  T3) 


Note:  X  moved  in  right  direction  and  met  target;  * 
**metric  worsened  over  time  but  target  was 


metric  improved  over  time  but  did  not  achieve  target  e.g.  green,  85%,  or  95%; 
met  during  the  timeframe 
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Relationship  of  current  findings  to  previous  findings 

In  March  2010,  the  Army  Nurse  Corps  began  implementing  a  professional  practice  model  to 
improve  clinical  practice,  quality  of  care,  patient  outcomes,  and  solidify  a  culture  of  evidence- 
based  practice  across  the  Corps,  focusing  on  inpatient  units  first.  At  the  time  this  project  began 
evidence  from  the  literature  (American  Association  of  Colleges  of  Nursing,  2002;  Hoffart  & 
Woods,  1996;  Institute  of  Medicine,  2003;  Aiken,  Clarke,  &  Sloane,  2008;  Landmark,  2008)  and 
data  from  Tripler’s  inpatient  units  (A.I.Corulli,  personal  communication,  August  2009)  indicated 
that  professional  practice  models  improved  staff  satisfaction  and  patient  outcomes.  Over  half 
the  metrics  tracked  in  this  project  improved  over  time  or  achieved/maintained  target  goals  by 
time  three.  While  these  results  were  not  as  robust  as  hoped  for,  both  clinics  made  progress 
overall  in  the  right  direction.  When  collapsed  over  time,  FM  metrics  were  at  target  more  often 
then  IM.  This  is  consistent  with  the  literature  in  that  professional  staff  satisfaction  is  associated 
with  more  positive  patient  outcomes. 

Effect  of  problems  or  obstacles  on  the  results; 

Military  and  personal  obligations  affected  data  collection  and  analysis.  Between  T1  and  T2, 
COL  Trego  moved  to  Germany  and  LTC  Siaki  was  deployed.  LTC  Siaki  re-deployed 
immediately  prior  to  T3  data  collection  of  the  EOM  II®  and  PSQ-18.  In  June  2012,  before  T3 
data  for  HEDIS,  ICE,  and  APLSS  was  available,  LTC  Siaki  and  Ms.  Byrne  relocated  to  Joint 
Base  Lewis-McChord  and  Arizona  respectively.  Coordinating  work  schedules  and  availability 
across  five  time  zones  (Boston,  Hawaii,  Germany,  Arizona,  and  Washington)  delayed  qualitative 
data  analysis  completion  by  more  than  six  months. 

Two  major  events  affected  this  project  that  were  beyond  the  influence  of  study  personnel. 
First,  the  Army  began  changing  over  to  a  medical  home  model  (PCMH).  Changing  to  a  PCMH 
involved  restructuring  staffing  into  teams,  realigning  positions  with  duties/responsibilities 
described  in  the  PCMH  model,  physical  construction  to  the  clinic  layout,  notifying  patients  of  the 
changes  in  providers  and  the  way  business  was  being  conducted  (e.g.  what  to  expect  from  a 
PCMH),  taking  on  the  arduous  process  of  becoming  a  certified  PCMH,  and  opening  outlying 
clinics.  Philosophical  underpinnings  of  both  the  PCMH  and  PPM  focus  on  better  patient 
outcomes  by  optimizing  care  and  professional  practice.  Theoretically,  the  two  models  are  quite 
compatible.  Clinically  however,  challenges  associated  with  implementing  both  systems 
simultaneously  introduced  significant  change  and  turbulence  to  the  clinics. 

Second,  the  PPM  also  changed  in  response  to  the  ANC’s  efforts  to  both  improve  the  PPM 
and  provide  Army  nurses  with  an  overarching  philosophy  of  professional  practice  that  informed 
the  full  spectrum  of  nurses  from  novice  to  expert,  bedside  clinician  to  commanders.  The  PPM 
was  no  longer  the  central  focus  but  part  of  an  overall  philosophy  of  care.  The  PPM  name 
changed  to  the  Patient  CaringTouch  System  (PCTS),  EBP  was  re-emphasized,  and  the  design 
of  the  elements  of  the  PPM  changed.  For  example,  the  concept  of  shared  governance  and  a 
lead  RN  was  retained,  but  roles,  responsibilities,  job  descriptions,  and  charters  changed  several 
times.  Currently  the  PCTS  is  in  sustainment  mode. 

Other  challenges  faced  by  staff  that  potentially  affected  staff  satisfaction  included  frequent 
leadership  changes,  staff  turnover,  and  financial  constraints  including  a  hiring  freeze.  For 
example,  FM  had  two  different  Clinical  Nurse  Officers  in  Charge  (CNOIC)  while  IM  had  four 
different  CNOIC’s  over  the  three  data  collection  points.  Both  clinics  were  authorized  nursing 
positions  but  were  unable  to  fill  them  in  a  timely  manner  due  to  a  system  wide  hiring  freeze. 
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Qualitative  data  reported  earlier,  support  these  observations.  Specifically,  staff  commented  on 
change,  communication,  and  lack  of  resources  such  as  adequate  staff,  all  of  which  are 
recognized  elements  of  work  turbulence  (Jennings,  2008).  Internal  Medicine  clinic  experienced 
more  turbulence  than  FM,  possibly  explaining  the  lower  EOM  II®  scores  in  IM.  Encouragingly, 
both  clinics  improved  or  maintained  at  least  50%  of  metrics  in  target. 

Several  other  issues  also  affected  the  study.  Two  surveys,  APLSS  and  ICE  are  collected  at 
the  MEDCOM  level  and  reported  back  to  the  clinic  in  quarterly  intervals.  Results  usually  are  1  or 
2  quarters  behind.  For  example,  data  collected  in  January  may  not  be  available  to  the  clinics 
until  June  or  September.  When  T3  ICE  data  were  received  from  the  organization,  more  than  six 
months  had  passed.  By  then,  it  was  too  late  to  encourage  completion  of  the  surveys  in  IM.  Both 
clinic  nursing  leadership  and  the  department  that  collates  and  distributes  the  results  were 
contacted  to  verify  the  low  numbers.  The  number  of  ICE  surveys  received  at  T 1  (FM  n=1350;  IM 
n=618)  was  significantly  different  from  T2  (FM  n=1090;  IM  n=47)  and  T3  (FM  n=1 1086;  IM 
n=12). 

A  similar  problem  occurred  with  the  EOM  II®,  with  most  surveys  collected  at  T1  (FM  n=18; 

IM  n=24)  compared  with  T2  (FM  n=14;  IM  n=19)  and  T3  (FM  n=12;  IM  n=7).  Study  personnel, 
Ms.  Byrne  in  particular,  recruited  several  times  in  each  time  period,  stressed  the  anonymous 
nature  of  the  survey,  engaged  nursing  leadership  at  each  clinic,  increased  study  personnel 
presence  in  the  clinics,  sent  weekly  emails  to  key  personnel  in  each  clinic,  reassessed  data 
collection  processes  in  an  iterative  fashion,  reviewing  with  project  mentor  Dr.  Lusardi  as 
needed,  and  provided  refreshments  during  the  collection  times  however  no  further  surveys  were 
collected.  On  the  other  hand,  staff  from  both  clinics  sought  out  study  personnel  asking  to  speak 
anonymously.  These  comments  were  similar  to  written  comments  in  that  leadership  style, 
communication,  change,  ineffectiveness  of  UPCs,  and  feeling  as  if  they  were  not  practicing  to 
their  full  potential  influenced  job  satisfaction  levels  and  morale. 

One  item  had  a  neutral  effect  but  did  require  an  amendment  during  the  study.  Open 
communication  was  maintained  from  pre-submission  development  through  analysis  with  the 
statistician.  In  collaboration  with  the  statistician  Dr.  Glaser,  a  repeated  measures  design  was 
originally  planned.  However,  this  was  not  feasible  due  to  the  anonymous  nature  of  the  three 
patient  surveys.  LTC  Siaki  and  COL  Trego  discussed  this  on  several  more  occasions  with  Dr. 
Glaser.  LTC  Moore  revisited  this  issue  with  Dr.  Glaser  after  LTC  Siaki  deployed.  The  design 
was  changed  to  a  2x3  factorial  design.  This  simplified  data  collection,  analysis,  and  retained  the 
evidence-based  nature  of  the  project.  Trying  to  track  contact  information  of  participants  for  a 
repeated  measures  design  would  have  necessitated  approval  from  an  Internal  review  Board  to 
collect  personally  identifiable  information  and  delayed  the  project. 

Limitations 

Limitations  were  identified  prior  to  and  during  this  evidence-based  project.  Known  limitations 
included  the  total  sample  size  of  RNs  in  both  clinics  with  LPNs,  medics,  and  medical  technicians 
making  up  the  majority  of  the  nursing  staff,  the  impending  implementation  of  the  PCMH, 
frequent  staffing  turnover  due  to  military  operations,  and  availability  of  staff  satisfaction  surveys 
developed  for  multiple  levels  of  nursing  staff  in  ambulatory  clinics.  Faculty  at  a  TSNRP 
sponsored  grant  camp  recommended  the  revised  Nursing  Work  Index  (NWI)  to  measure  staff 
satisfaction  rather  than  the  planned  instrument.  When  the  authors  were  contacted  they 
recommended  using  the  updated,  revised  instrument  renamed  the  Essentials  of  Magnetism  II 
(EOM-II©).  This  instrument  has  been  used  with  RNs  on  inpatient  and  outpatient  units  but  not 
with  other  levels  of  nursing  staff.  However  the  team  decided  to  use  the  EOM-II©  for  several 
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reasons:  1)  Recommendation  of  TSNRP  grant  camp  faculty;  2)  A  relative  lack  of  comparable 
instruments  for  non-RN  nursing  staff;  3)  The  EOM-II®  focus  on  professional  practice;  and  4) 

The  ANC  recognizes  the  important  contributions  to  patient  care  from  each  area  of  it’s  triangle: 
Officers  (RNs),  Enlisted  (LPNs,  medics,  and  other  unlicensed  personnel),  and  civilians  (all 
levels).  Midway  through  the  project,  study  personnel  reviewed  the  literature,  contacted  the 
EOM-II©  authors,  and  the  American  Nurses  Credentialing  Center  (ANCC).  As  before,  no  other 
instrument  measuring  professional  job  satisfaction  for  different  levels  of  nursing  staff  were 
identified. 

Limitations  identified  during  the  course  of  the  project  included  a  delay  in  forward  momentum 
in  both  clinics  of  optimal  functioning  of  the  UPC  due  to  changes  in  leadership  focus  and 
personnel,  the  emphasis  on  implementation  of  the  PCMH,  changes  in  the  metrics  (APLSS 
questions  and  pneumovax),  the  psychometric  properties  of  the  PSQ-18,  and  low  sample  sizes 
at  T2,  and  T3  for  multiple  metrics. 

Psychometric  properties  of  the  PSQ-18  are  concerning  in  that  the  highest  coefficient  alpha 
was  obtained  for  Technical  Quality  (a  =  .709)  followed  by  Accessibility  and  Convenience  (a  = 
.689).  The  rest  were  <  .623.  Between  T2  and  T3,  LTC  Siaki  and  Ms.  Byrne  RA,  revisited  the 
PSQ  literature.  Two  studies  that  also  altered  or  deleted  subscales  of  the  PSQ-18  reported  low 
psychometrics  (Ricci,  2012;  Stewart,  Kroth,  Schuyler,  Bailey,  2010)  as  noted  in  table  7.  Given 
that,  in  part,  alpha  is  a  function  of  the  number  of  items;  unless  there  is  a  substantive  correlation 
between  two  items,  a  two-item  scale  will  tend  to  manifest  lower  reliability  scores.  After 
reviewing  the  literature  and  conferring  with  the  statistician,  an  ad-hoc  analysis  using  exploratory 
factor  analysis,  reflected  a  two  factor  solution  similar  to  that  obtained  by  Rubio,  Pearson,  Clark, 
and  Breitkopf  (2007).  Specifically,  the  items  aligned  along  a  positive  or  negative  axis,  indicating 
satisfaction  or  dissatisfaction. 

Implications  for  these  values  is  that  in  the  context  of  significance  testing,  measurement  error 
may  compromise  the  power  of  the  analysis  (hence,  culminating  in  inflated  Type  II  error  =  false 
negative  findings).  Additionally,  there  are  no  hard  and  fast  rules  for  what  constitutes  an 
"acceptable"  reliability  score.  As  it  depends  upon  the  use  and  intent  of  the  instrument/scale, 
many  have  cited  .7  as  acceptable  (D.  Glaser,  personal  communication,  August  2012). 

However,  again,  .7  may  be  problematic  if  the  intent  is  to  use  the  instrument  for  individual 
decisions  (in  that  case  an  a  of  .8  may  be  deemed  to  be  minimally  acceptable)  However,  the 
intent  of  this  project  was  to  improve  overall  satisfaction,  access,  and  continuity  in  two  local 
ambulatory  clinics. 
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Table  7 


Reliability  values  from  selected  studies  using  a  modified  PSQ-18 

Coefficient  Alpha 

TSNRP:  HU0001 

-10-1-TS18 

Marshall* 

Stewart 

Ricci, 

Siaki 

Standardized 

Average  Item 

et  al  1 994 

et  al  2009 

2012 

2o’l2 

Alpha 

Intercorrelation 

General 

Satisfaction 

.75 

.58 

.59 

.62 

.632 

.462 

PSQ-18  subscales 

Technical  Quality 

.74 

.77 

.57 

.71 

.718 

.389 

Interpersonal 

Manner 

.66 

.57 

.48 

.44 

.449 

.289 

Communication 

.64 

.64 

.27 

.45 

.478 

.314 

Time  Spent  with 
Doctor/Provider 

.77 

.67 

.76 

.56 

.562 

.390 

Accessibility  / 
Convenience 

.75 

N/A 

.57 

.69 

.695 

.362 

Financial 

.73 

N/A 

.67 

N/A 

N/A 

N/A 

Note:  *  original  developer  of  PSQ-18 


Conclusion 

Professional  practice  models  facilitate  staff  satisfaction,  the  design  and  delivery  of  optimal 
care,  and  improve  patient  outcomes  (AACN,  2002;  Aiken,  Clarke,  &  Sloane,  2008;  Hoffart  & 
Woods,  1996;  Girard,  Linton,  &  Besner,  2005).  Family  and  Internal  Medicine  outpatient  clinics  at 
Tripler  Army  Medical  Center  implemented  a  nursing  PPM  guided  by  the  Iowa  Model  of 
Evidence-Based  Practice  during  2009-2012.  To  this  team’s  knowledge,  this  is  the  first  EBP 
project  looking  at  outcomes  associated  with  a  military  PPM.  Although  results  from  this  project 
were  not  as  robust  as  those  reported  in  civilian  hospitals  and  inpatient  settings,  they  were 
encouraging  nonetheless.  Essential  elements  of  the  PPM  identified  in  the  literature  and  by 
subject  matter  experts  were  readily  apparent  despite  significant  work  turbulence.  For  example, 
despite  limitations  in  space,  equipment,  staffing,  and  time,  nursing  staff,  guided  by  the  PPM, 
continued  to  focus  on  access,  quality  of  care,  and  communication;  demonstrating  their 
commitment  to  professional  values  and  ethics  and  interdisciplinary  collaboration.  Qualitative 
data  also  indicated  that  nursing  staff  recognized  leadership  support  and  the  importance  of  the 
UPCs/shared  governance  to  quality  practice,  despite  feeling  overworked  and  underappreciated. 

Results  from  this  project  align  with  evidence  in  the  literature  regarding  the  importance  and 
effect  of  staff  satisfaction  on  patient  satisfaction,  quality  care,  and  outcomes.  Negative  effects  of 
work  turbulence  e.g.  military  operations,  changes  to  the  PPM,  the  structure  and  function  of  the 
clinics  themselves,  and  implementation  of  the  PCMH,  emerged  in  all  metrics.  Although  this 
EBP  project  was  not  designed  to  test  a  hypothesis  or  produce  generalizable  knowledge, 
opportunities  for  improvement  emerged  that  are  worth  noting. 

One,  nurse-sensitive  optimal  care  indicators  and  outcome  measures  for  outpatient  clinics 
are  not  as  well  defined  as  inpatient  metrics  although  the  ANA  and  the  American  Academy  of 
Ambulatory  Care  Nursing  (AAACN)  have  workgroups  currently  focusing  on  this  issue.  Duties 
and  responsibilities  of  nursing  staff  in  a  PCMH  differ  somewhat  from  traditional  outpatient 
settings.  Reasons  for  high  staff  turnover  rates  and  timely  replacements  differ  in  military  versus 
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civilian  organizations.  Thus  the  outcome  measures  chosen  for  this  project,  while  practical  may 
not  have  been  the  best  fit  in  this  context. 

Two,  the  ANC  actively  promotes  nursing  staff  at  all  levels  to  work  to  the  full  extent  of  their 
training  and  education.  The  majority  of  work  regarding  professional  practice  and  patient 
outcomes  has  been  focused  on  inpatient  RNs  in  civilian  settings.  More  research  and  EBP 
projects  are  needed  to  define  nurse-sensitive  optimum  care  indices  for  military  and  civilian 
ambulatory  clinics  and  identify  important  contributions  of  non-RN  staff  in  both  traditional  and 
PCMH  models.  Instruments  to  assess  healthy  work  environments  for  non-RN  staff  is  another 
area  of  opportunity. 

Three,  the  ANCs  PPM  has  matured  into  the  PCTS,  a  comprehensive,  all  encompassing 
philosophical  approach  to  professional  practice.  As  noted  above,  research  and  EBP  projects 
evaluating  the  short  and  long  term  effects  of  this  significant  paradigm  shift  and  the  role  in 
models  of  care  such  as  the  PCMH  across  all  military  services,  would  enhance  the  ongoing 
efforts  to  design  and  deliver  optimum  care  to  military  beneficiaries  worldwide  (figure  8). 

Last,  significant  challenges  due  to  turbulence  from  multiple  sources  targets  the  need  for 
education  and  training  for  leaders  and  staff  with  regards  to  change  management.  Likewise, 
education  and  training  targeting  PPM  elements:  effective  communication,  autonomy,  and 
interdisciplinary  collaboration,  in  the  spirit  of  the  lOM  report  on  nursing  would  be  beneficial. 

Figure  8:  Patient  and  Family  Centered  System  of  Care  and  Patient  CaringTouch  System 


28 


Principal  Investigator:  Siaki,  MAJ  Leilani 


USU  Project  Number:  N10-C04 


Significance  of  Study  or  Project  Resuits  to  Miiitary  Nursing 

In  1999  the  Institute  of  Medicine  (lOM)  published  a  sobering  report  outlining  grim  statistics 
related  to  the  delivery  of  suboptimal  healthcare.  One  key  strategy  recommended  by  the  lOM  to 
improve  the  quality  of  care  and  patient  outcomes  focused  on  translating  research  findings  into 
evidence-based  clinical  practice.  Florence  Nightingale  used  this  critical  approach  to  change  the 
professional  standing  of  nurses,  the  quality  of  nursing  care  delivered,  and  saved  lives.  A  decade 
or  more  before  the  lOM  report,  the  American  Nurses  Credentialing  Center  (ANCC)  validated  the 
importance  of  professional  practice  models  to  optimal  patient  outcomes.  These  models  provide 
the  scaffolding  that  shape  work  cultures,  support  professional  nursing  values,  improve  nurse 
satisfaction  and  retention,  patient  satisfaction.  For  more  than  100  years,  the  ANC  has 
maintained  a  commitment  to  professional  excellence  and  proactively  initiated  strategies  that 
facilitate  best  patient  outcomes.  From  this  perspective,  nursing  leaders  at  TAMC  systematically 
evaluated  the  literature  and  sought  expert  consultation  for  evidenced-based  professional 
practice  models.  A  PPM  that  emphasized  expert  differentiated  clinical  practice,  proactive 
leadership,  and  interdisciplinary  teamwork  was  successfully  implemented  on  the  inpatient  wards 
at  TAMC.  Based  on  results,  TAMC  began  implementing  the  PPM  in  outpatient  clinics,  a  first  in 
military  healthcare  facilities.  Despite  significant  organizational  and  clinic  level  turbulence,  both 
clinics  maintained  or  achieved  target  goals  for  about  50%  of  the  tracked  metrics.  While  many  of 
the  metrics  did  not  achieve  statistical  significance,  results  trended  in  the  direction  expected 
based  on  current  evidence  regarding  effects  of  professional  practice  models.  Overall  goals  of 
EBP  projects  grounded  in  evaluation  science  are  to  assess  strengths  and  weaknesses  of  locally 
implemented  processes  or  projects.  No  attempt  at  creating  generalizable  knowledge  was  made. 
Pilot  testing  of  EBP  projects  prior  to  unit  or  system-wide  implementation  is  encouraged.  Based 
on  that  perspective,  this  project  was  successful  in  that  several  opportunities  for  improvement 
surfaced  and  both  EBP  and  research  recommendations  can  be  offered.  Specific  EBP 
recommendations  include: 

1 .  Now  that  the  PCTS  is  in  sustainment  phase  across  the  ANC,  continue  to  track  and 
analyze  currently  identified  nurse-sensitive  quality  of  care  and  outcomes  data  locally  and 
at  selected  facilities  system  wide.  This  data  can  be  used  to  identify  specific  areas 
needing  improvement  related  to  the  PCTS,  the  local  organization,  or  the  system-at-large. 

2.  Clarify  the  synergistic  nature  of  the  PCMH  and  the  PCTS  for  both  staff  and  leadership 
and  implement  programs  to  manage  change. 

3.  Differentiated  practice  levels  within  both  the  PCMH  and  the  PCTS  continue  to  need 
better  clarification  for  both  leaders  and  staff. 

4.  Quantitative  and  more  in-depth  qualitative  data  in  the  form  of  focus  groups  will  provide 
the  necessary  detail  needed  to  achieve  the  previous  three  recommendations. 

5.  Data  from  these  recommendations  can  be  used  to  develop  similar  PCTS  models  for 
military  nurses  across  the  services. 

Research  recommendations  include: 

1 .  Develop/modify  an  instrument  to  measure  professional  job  satisfaction  for  all  levels  of 
nursing  staff  consistent  with  the  current  composition  of  the  ANC.  This  instrument 
ultimately  would  be  useful  across  military  services  and  in  other  healthcare  organizations 
that  rely  on  non-RN  care  providers. 

2.  Evaluate  the  effects  of  PCTS  using  a  multilevel  modeling/  nested  hierarchal  statistical 
model  to  assess  non-provider  (RN,  LPN,  medics)  team  effects  on  currently  known 
ambulatory  quality  of  care,  access,  continuity,  and  satisfaction  indices. 
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3.  Identify/validate/develop  nursing-sensitive  indicators  of  ambulatory  quality  of  care, 
access,  continuity,  and  satisfaction  indices  that  will  improve  patient  outcomes  for  all 
military  beneficiaries  irrespective  of  services  similar  to  the  MILNOD  data. 

4.  Validate  the  modified  PSQ-18  for  all  military  beneficiaries  service  immaterial  and  other 
similar  populations  for  which  a  modified  instrument  (e.g.  deleting  the  financial  subscales) 
is  more  appropriate. 

Since  this  project  began,  the  PPM  has  evolved  into  an  overarching  philosophy  of  care  and 
implemented  across  all  ANC  venues.  These  are  the  first  known  reported  results  of  the  effects  of 
the  PPM  on  nurse-sensitive  staff  and  patient  outcome  measures.  Projects  such  as  this  highlight 
the  importance  of  EBP  and  research  efforts  to  improve  practice,  facilitating  nursing  staff  at 
multiple  levels  to  work  at  the  top  of  their  scope,  to  make  sustained  progress  to  achieve  optimal 
outcomes  for  patients,  to  continue  the  critical  work  Nightingale  started  over  100  years  ago;  to 
sustain  the  ANC  mission  of  fostering  excellence  in  military  nursing  care. 
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Changes  in  Clinical  Practice,  Leadership,  Management,  Education,  Policy,  and/or  Military 

Doctrine  that  Resulted  from  Study  or  Project 

To  date,  results  of  this  EBP  project  have  not  produced  local  or  system-wide  changes. 
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Developed  a  cell  line 

Developed  a  tissue  or 
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Table  7 

Recruitment  and  Retention  Tabie 


Recruitment  and  Retention  Aspect 

Number 

Staff 

Number 

Patients 

FM 

IM 

FM  IM 

Subjects  Projected  in  Grant  Application 

20-25  20-25 

500/week 

Subjects  Available 

17 

24 

500/week 

Subjects  Contacted  or  Reached  by  Approved  Recruitment  Method 

T1 

17 

24 

200  250 

T2 

14 

19 

250  250 

T3 

12 

7 

300  250 

Subjects  Screened 

19 

24 

— 

Subjects  Ineligible 

0 

0 

0  0 

Subjects  Refused/unavailable  i.e. 
military  training 

leave, 

T1 

0 

0 

T2 

4 

2 

N/A 

T3 

5 

13 

Human  Subjects  Consented 

Completed  survey  indicated  consent  to  participate 

Subjects  Who  Withdrew 

N/A  -  anonymous  survey 

Subjects  Who  Completed  Study 

T1 

17 

24 

160  213 

T2 

14 

19 

194  211 

T3 

12 

7 

228  214 

Subjects  With  Complete  Data 

57-89 

1153-1184 

Subjects  with  Incomplete  Data 

EOM 

Question  with  the  most  missing  data  was  level  of 
education  (n=37).  All  other  questions  n=0-5. 

PSQ 

Varied  by  question 

-  n=36  -  67(3  -  5.5%) 

APLSS/ICE 

No  missing  data  -  received  in  aggregate  format 
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Table  8 

Demographic  Characteristics  of  the  Sampie 


Characteristic 

N  (%) 

Staff  -  Clinic  FM 

44  (46.8%) 

IM 

50  (53.2%) 

Highest  level  of  education* 

MSN 

1  (2.7%) 

BSN 

10  (27%) 

Diploma 

6  (16.2%) 

ADN 

4  (10.8%) 

Other 

15  (40.5%) 

Years  of  experience* 

n=32,  6.97  years  (SD  =  7.15) 

Shift  worked*  8/1 0  hour  days 

34  (89.5%) 

8/10  evenings 

4  (10.5%) 

Patients 

PSQ** 

Clinic  FM 

582(47.7%) 

IM 

638  (52.3%) 

Time  1 

373  (30.6%) 

Time  2 

405  (33.2%) 

Time  3 

442  (36.2%  ) 

Times  seen  in  clinic  past  6  months 

1-2 

491  (43.7%) 

3-4 

393  (35%) 

5  or  more 

239  (21.3%) 

Missing 

97  (8%) 

Know  assigned  healthcare  team 

Yes 

798  (69.9%) 

No 

359  (31%) 

Missing 

63  (5.2%) 

Know  PCM’s  name 

Yes 

1037(89%) 

No 

128  (11%) 

Missing 

55  (4.5%) 

Know  primary  nurse’s  name 

Yes 

302  (26.3%) 

No 

847  (73.7%) 

Missing 

71  (5.8%) 

*Time  one  data  only  as  multiple  staff  completed  surveys  at  all  three  time  periods 
**  Aggregated  across  time 
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